Convexification and Deconvexification for Training Neural Networks

This paper presents a new method of training neural networks including deep learning machines, which is based on the idea of convexifying the training error criterion by the use of the risk-averting error (RAE) criterion. Convexification creates tunnels between the depressed regions around saddle points, tilts the plateaus, and eliminates nonglobal local minima. The difficulties in computing the RAE and its gradient and in selecting the value of its risk-sensitivity index λ are eliminated with the normalized RAE (NRAE). The new method, called gradual deconvexification (GDC), starts with the NRAE with a very large λ, gradually decreases it, and switches to the RAE as soon as the RAE becomes computationally manageable. This way, the gradients of the plateaus in the training error criterion are effectively but not excessively raised. Numerical experiments show the effectiveness of GDC as compared with unsupervised pretraining, which is the state of the art in training deep learning machines. After the minimization process is terminated by crossvalidation, a statistical pruning method is used to enhance the generalization capability of the resultant neural network. Numerical results show further reduction of the testing criterion.